Guide to the Basics of Software Testing Quality Metrics

The best way to control progress in the entire QA process is to use at-work software testing metrics.

👉 Determine desired performance during the planning stage and compare them with actual results obtained by the testing team.

– Do the numbers match?

😀 Congratulations!

– Your development and testing efforts will allow you to create high-quality software products.

🤔 Are the results obtained far from the standard ones?

– Test effectiveness on the project requires some improvement…

Thus today, we will discuss the Main Software Test Metrics used in the QA process in modern Agile teams.

Also read to this subject:

• Key Software Testing Metrics & KPI’s
• What are Quality Gates and How Will They Help Your Project?

Test Metric Concept

Software test metrics quantify various aspects of the test process, including productivity, reliability, and effectiveness. It is advisable to use them at all stages of the Software Development Life Cycle (SDLC), as this will allow you to identify areas for improvement timely, make informed decisions about the project, and achieve the desired software quality.

Each metric in the process of developing a digital solution goes through a particular life cycle, which consists of four stages:

• Analysis. First, you need to select the testing metrics used on the project and determine the desired results.
• Communication. Ensuring all process participants understand why and how to use this metric is essential.
• Quantification. At this stage, an assessment of the testing process is performed according to selected criteria.
• Drawing up reports. Once the assessment results are received, a detailed report should be compiled and distributed to the members of the QA team and the dev team.

Regardless of the type, these test metrics cycle stages are common to all metrics. What are they like? Let’s consider the ones below ⬇️

Classes and types of software testing quality metrics

All indicators used for software test measurement are traditionally divided into four classes, depending on which aspect testing efforts help to evaluate:

• Process metrics – allow us to draw conclusions about QA process efficiency and indicate to the testing team which part of the software testing process needs improvement.
• Product metrics – metrics that provide information about product quality. For example, you can consider the total number of defects or results of performance testing.
• Project metrics – identifiers that allow you to evaluate how well the test team works. Define test automation or test execution coverage, track fixed defects percentage, and you will understand what your project is missing.
• People metrics – indicators that help assess the qualifications of an individual test team member. For example, you can determine how many bugs account for one specialist or the percentage of defects identified by the tester.

In addition to dividing into classes, there is another way to systematize metrics. Among all existing software testing metrics, distinguish:

• Absolute metrics
• Derivative metrics

The first ones are absolute numbers, which are easy to measure: for example, the number of bugs found, total number of requirements and number of requirements covered, number of test cases, and quantity passed and failed test cases.

Derivative metrics are indicators obtained as a result of combining several absolute metrics. For example, you can calculate the percentage of all and passed test cases, test coverage, defect density, etc.

Let’s look at both types of software testing metrics in detail ⬇️

Absolute Test Metrics as a Basis For Software Test Measurement

Measured absolute numbers that can be used to calculate work-relevant testing team metric include:

→ total number of test cases written by the QA team;
→ Quantity passed, failed, blocked test cases;
→ number of defects – found, critical, accepted, rejected, deferred;
→ planned time on software testing and actual test hours number;
→ number of requirements for software products;
→ total number of bugs detected after delivery;
→ test execution time.

This is not a full list of absolute test metrics used in modern manual and automated testing. All of the above indicators have one thing in common: they can be used to calculate derived metrics, which show an accurate picture of the condition of the QA process on the project.

Derivative metrics and methods for their calculation

Getting derived metrics, or calculated metrics, is not as simple as getting absolute numbers – you need to use specific formulas. We list the most commonly used indicators and methods for calculating each.

Metrics, Which Measure Testing Efforts

These software testing metrics can be attributed to the following:

In this formula:

• Total number of tests is the total count of tests executed.
• Actual time spent testing is the duration of time over which the tests were run.

If you want to express this in terms of tests per hour, you can specify the unit of time accordingly. For example, if the actual time spent testing is in hours, the result will be the number of tests run per hour.

In this formula:

• Number of test cases is the total number of test cases developed.
• Time spent developing is the total time spent in developing those test cases.

This will give you the number of test cases developed per unit of time (e.g., test cases per hour).

In this formula:

• Number of tests reviewed is the total number of test cases reviewed.
• Time spent reviewing is the total time spent reviewing those test cases.

This will give you the number of test cases reviewed per unit of time (e.g., tests reviewed per hour).

In this formula:

• Total number of bugs is the total count of defects or bugs found.
• Total number of tests is the total count of test cases executed.

This will give you the average number of defects found per test case.

Indicator That Allows You to Evaluate Test Effectiveness

These metrics demonstrate the quality of the test set. In other words, it shows how many bugs you can found using test cases. It is calculated using the formula:

In this formula:

• Number of defects detected with one test is the number of defects found during testing.
• Total number of bugs is the total count of defects detected during the testing phase.
• Number of bugs found after release is the number of defects discovered post-release.

This formula calculates the percentage of defects detected during testing relative to the total number of defects (both during testing and after release).

Test Coverage Metrics

Test coverage means the amount of testing performed by a particular test set. Test coverage metrics measure:

In this formula:

• Number of completed test runs is the count of test runs that have been executed.
• Number of planned test runs is the total count of test runs that were planned to be executed.

This formula gives you the percentage of planned tests that have been completed.

In this formula:

• Number of requirements covered is the count of requirements that have been successfully covered by test cases.
• Total number of requirements is the total count of requirements specified for the project.

This formula gives you the percentage of requirements that are covered by test cases.

Test Economics Metrics

Such test metrics allow the test team to estimate the cost of testing, stay within the project budget, correctly plan the choice of infrastructure and tools, and determine the required number of quality assurance specialists. These include:

• Total Allocated Costs – budget approved by QA directors on test activities within a specific project or time interval.
• Actual cost – the amount spent on the test process. To calculate this metric, use the data about expenses per test hour. Per test hour, test case, or requirements.
• Budget Variance – the difference between planned and actual expenses for the testing process.
• Time Variance – the difference between planned and actual time testing.
• Cost per bug fix – error correction cost per team member.
• Cost of not testing – price of actual rework efforts spent due to insufficient testing of new functionality.

Test Team Metrics

These indicators reflect the uniform load distribution on each team member and the work efficiency of the QA team. It is advisable to calculate:

• Quantity of defects returned per team member;
• Quantity of valid defects, which are subject to re-testing by each team member;
• Number of test cases assigned to one specialist;
• Quantity of test cases executed by team members.

Scroll software testing metrics may vary depending on your project’s characteristics and goals. Choosing metrics that meet your needs is essential for the right planning and implementing test processes and achieving desired results. Next, we will discuss the correctness test metrics in QA.

How to Select the Right Software Testing Metrics?

When choosing metrics for testing process assessment, it is necessary to pay attention to specific criteria:

1. Test metrics should be easy to measure. All metrics discussed in this article are easy to calculate. For example, let’s say we need to determine the percentage of critical defects on a project, even though the general number of defects is 45, and the critical ones are 10. Using the formula given above, we get: (10 / 45) * 100 = 22 %
2. Correct QA metrics are easy to update according to project changes. This is especially true for automated testing, which requires regular test updates.
3. Correct testing metrics help you achieve your business goals. For example, if cutting your budget is essential to you, consider test economics metrics. Teams looking to speed up future releases should focus on assessing test effectiveness and productivity of testing teams.

Determine the indicators you need at the very beginning of a project. This will allow you to optimize your testing efforts, stay within budget, and meet deadlines.

Recommendations for Selection of Software Test Metrics

Use our tips to choose optimal metrics for your testing process:

• Consider the type of testing that predominates in your project. For example, if you test a digital solution manually, it makes sense to focus on tracking defect metrics. If most of your tests are automated, pay attention to the percentage of passed test cases or actual test hours number.
• Agree on testing metrics with the development methodology. Most teams use the agile development methodology. It makes sense for them to concentrate on the number of defects identified after the next sprint or at production. If your team works on a waterfall model, test coverage metrics and other indicators will be critical for you.
• Use metrics important to everyone. Tracking test coverage or a number of defects is indispensable for the QA and development team; however, this data will not be very informative for the customer. So, balance the metrics indicative for the team with test economics metrics or indicators that show time for future releases.
• Limit the amount of test metrics you see. Don’t try to control everything at once. Even at the planning stage, determine the most significant indicators to you and track them throughout your work on the project.
• Share responsibilities. Every test team member must understand which indicators are in their area of ​​responsibility. The project manager can control test coverage metrics or team productivity, and the QA lead can take over tracking defect removal efficiency and similar metrics that will help in future test planning.

Such a thorough approach to tracking key testing metrics will benefit your team, which we will discuss next.

Benefits of Using Test Metrics on Your Project

Teams that track important metrics on their projects notice clear improvements in the testing process, namely:

• Increase of testing system’s efficiency. Systematic tracking of selected test metrics allows the team to identify areas of improvement and optimize the testing process.
• Quality improvement of the software product. Metrics help identify bugs in the early stages of development and ensure the high quality of the final solution, which contributes to increased end-user satisfaction.
• Reducing risks by project. Analyzing reliable data obtained by tracking testing metrics allows you to predict testing progress, identify problems promptly, and thus effectively manage existing risks.
• Continuous improvement of the testing team. Tracking metrics allow the team to draw conclusions regarding their productivity and determine the factors that provoked the occurrence of bottlenecks. Such work process analysis will enable you to constantly look for new work methods and tools if you obtain unfavorable results.

Thus, software testing metrics allow you to determine the level of testing process efficiency based on quality data, software product, and team productivity. If tracking metrics show that you have not achieved the desired results, change your testing strategy, try new tools, and work with your team. This testing approach will optimize your test process on the project and help you create high-quality digital solutions.